Neural Network Processor for Handling Differing Datatypes

ABSTRACT

Embodiments relate to a neural engine circuit that includes an input buffer circuit, a kernel extract circuit, and a multiply-accumulator (MAC) circuit. The MAC circuit receives input data from the input buffer circuit and a kernel coefficient from the kernel extract circuit. The MAC circuit contains several multiply-add (MAD) circuits and accumulators used to perform neural networking operations on the received input data and kernel coefficients. MAD circuits are configured to support fixed-point precision (e.g., INT8) and floating-point precision (FP16) of operands. In floating-point mode, each MAD circuit multiplies the integer bits of input data and kernel coefficients and adds their exponent bits to determine a binary point for alignment. In fixed-point mode, input data and kernel coefficients are multiplied. In both operation modes, the output data is stored in an accumulator, and may be sent back as accumulated values for further multiply-add operations in subsequent processing cycles.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/971,868, filed May 4, 2018, which is incorporated by reference in itsentirety.

BACKGROUND 1. Field of the Disclosure

The present disclosure relates to a circuit for instantiating neuralnetworks and more specifically to handing neural network operationsusing different types of data.

2. Description of the Related Arts

An artificial neural network (ANN) is a computing system or model thatuses a collection of connected nodes to process input data. The ANN istypically organized into layers where different layers perform differenttypes of transformation on their input. Extensions or variants of ANNsuch as convolution neural network (CNN), deep neural networks (DNN),recurrent neural networks (RNN) and deep belief networks (DBN) have cometo receive much attention. These computing systems or models ofteninvolve extensive computing operations including multiplication andaccumulation. For example, CNN is a class of machine learning techniquethat primarily uses convolution between input data and kernel data,which can be decomposed into multiplication and accumulation operations.

Depending on the types of input data and operations to be performed,these machine learning systems or models can be configured differently.Such varying configuration would include, for example, pre-processingoperations, number of channels in input data, kernel data to be used,non-linear function to be applied to convolution result, and applying ofvarious post processing operations. Using a central processing unit(CPU) and its main memory to instantiate and execute machine learningsystems or models of various configuration is relatively easy becausesuch systems or models can be instantiated with mere updates to code.However, relying solely on the CPU for various operations of thesemachine learning systems or models would consume significant bandwidthof a central processing unit (CPU) as well as increase the overall powerconsumption.

SUMMARY

Embodiments relate to a neural engine circuit that includes an inputbuffer circuit, a kernel extract circuit, and a multiply-accumulator(MAC) circuit. The input buffer circuit broadcasts input data to the MACcircuit. The kernel extract circuit sends kernel coefficients to the MACcircuit. The MAC circuit includes multiply-add (MAD) circuits andaccumulator circuits used to perform neural networking operations on thereceived input data and kernel coefficients. Each MAD circuit includes amultiplier, a shift register, an adder, an exponent adder, and a shiftoffset register. Each MAD circuit supports a fixed-point precision ofoperands (e.g., INT8) and a floating-point precision of operands (e.g.,FP16).

In one embodiment, each MAD circuit in a fixed-point operation modeoperates on fixed-point input data and kernel coefficients using amultiplier and adder while turning off unused devices for powerconservation. Each MAD circuit in a floating-point operation modeseparates input data and kernel coefficients into integer bits andexponent bits. The integer bits of the input data are multiplied withthe integer bits of a kernel coefficient. A binary point position forthe product of the multiplied integer bits is determined by adding theexponent bits of the input data and kernel coefficient to a binary pointvalue. The multiplied integer is shifted into position using the shiftregister. In both operation modes, the processed values are stored in anaccumulator circuit, and may be sent back as feedback information forfurther multiply-add operations in subsequent processing cycles.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram of an electronic device, according to oneembodiment

FIG. 2 is a block diagram illustrating components in the electronicdevice, according to one embodiment.

FIG. 3 is a block diagram illustrating a neural processor circuit,according to one embodiment.

FIG. 4 is a block diagram of a neural engine in the neural processorcircuit, according to one embodiment.

FIG. 5 is a conceptual diagram illustrating loops for processing inputdata at the neural processor circuit, according to one embodiment.

FIG. 6 is a conceptual diagram illustrating segmenting the input datainto slices, tiles and work units, according to one embodiment.

FIG. 7 is a diagram illustrating programming of rasterizers incomponents of the neural processor circuit, according to one embodiment.

FIG. 8 is a flowchart illustrating a method of processing input data ina neural processor circuit, according to one embodiment.

FIG. 9A is a block diagram illustrating a multiply-accumulator (MAC)circuit in a fixed-point mode of operation, according to one embodiment.

FIG. 9B is a block diagram illustrating a MAC circuit in afloating-point mode of operation, according to one embodiment.

FIG. 10 is a diagram illustrating a process for multiplyingfloating-point input data and a kernel coefficient, according to oneembodiment.

FIG. 11 is a flowchart illustrating a method of performing multiply-addoperations in a MAC circuit, according to one embodiment.

FIG. 12 is a flowchart illustrating a method of processing fixed-pointinput data and kernel coefficients, according to one embodiment.

FIG. 13 is a flowchart illustrating a method of processingfloating-point input data and kernel coefficients, according to oneembodiment.

The figures depict, and the detail description describes, variousnon-limiting embodiments for purposes of illustration only.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,the described embodiments may be practiced without these specificdetails. In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

Embodiments of the present disclosure relate to a neural engine circuitfor performing neural network operations using input data and kerneldata in a fixed-point precision or a floating-point precision. Eachneural engine circuit includes an input buffer circuit, a kernel extractcircuit, and a multiply-accumulator (MAC) circuit. The MAC circuitincludes several multiply-add (MAD) circuits and several accumulatorcircuits. The MAD circuits are used for performing multiply-addoperations on input data and kernel coefficients having eitherfixed-point precision, floating-point precision, or both. The outputdata generated by the MAD circuits is stored in an accumulator circuitto be reused during subsequent processing cycles.

Exemplary Electronic Device

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as personaldigital assistant (PDA) and/or music player functions. Exemplaryembodiments of portable multifunction devices include, withoutlimitation, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. Other portable electronic devices,such as wearables, laptops or tablet computers, are optionally used. Insome embodiments, the device is not a portable communications device,but is a desktop computer or other computing device that is not designedfor portable use. In some embodiments, the disclosed electronic devicemay include a touch sensitive surface (e.g., a touch screen displayand/or a touch pad). An example electronic device described below inconjunction with FIG. 1 (e.g., device 100) may include a touch-sensitivesurface for receiving user input. The electronic device may also includeone or more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

FIG. 1 is a high-level diagram of an electronic device 100, according toone embodiment. Device 100 may include one or more physical buttons,such as a “home” or menu button 104. Menu button 104 is, for example,used to navigate to any application in a set of applications that areexecuted on device 100. In some embodiments, menu button 104 includes afingerprint sensor that identifies a fingerprint on menu button 104. Thefingerprint sensor may be used to determine whether a finger on menubutton 104 has a fingerprint that matches a fingerprint stored forunlocking device 100. Alternatively, in some embodiments, menu button104 is implemented as a soft key in a graphical user interface (GUI)displayed on a touch screen.

In some embodiments, device 100 includes touch screen 150, menu button104, push button 106 for powering the device on/off and locking thedevice, volume adjustment buttons 108, Subscriber Identity Module (SIM)card slot 110, head set jack 112, and docking/charging external port124. Push button 106 may be used to turn the power on/off on the deviceby depressing the button and holding the button in the depressed statefor a predefined time interval; to lock the device by depressing thebutton and releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Inan alternative embodiment, device 100 also accepts verbal input foractivation or deactivation of some functions through microphone 113. Thedevice 100 includes various components including, but not limited to, amemory (which may include one or more computer readable storagemediums), a memory controller, one or more central processing units(CPUs), a peripherals interface, an RF circuitry, an audio circuitry,speaker 111, microphone 113, input/output (I/O) subsystem, and otherinput or control devices. Device 100 may include one or more imagesensors 164, one or more proximity sensors 166, and one or moreaccelerometers 168. The device 100 may include components not shown inFIG. 1 .

Device 100 is only one example of an electronic device, and device 100may have more or fewer components than listed above, some of which maybe combined into a components or have a different configuration orarrangement. The various components of device 100 listed above areembodied in hardware, software, firmware or a combination thereof,including one or more signal processing and/or application specificintegrated circuits (ASICs).

FIG. 2 is a block diagram illustrating components in device 100,according to one embodiment. Device 100 may perform various operationsincluding image processing. For this and other purposes, the device 100may include, among other components, image sensor 202, system-on-a chip(SOC) component 204, system memory 230, persistent storage (e.g., flashmemory) 228, motion (orientation) sensor 234, and display 216. Thecomponents as illustrated in FIG. 2 are merely illustrative. Forexample, device 100 may include other components (such as speaker ormicrophone) that are not illustrated in FIG. 2 . Further, somecomponents (such as motion sensor 234) may be omitted from device 100.

Image sensor 202 is a component for capturing image data and may beembodied, for example, as a complementary metal-oxide-semiconductor(CMOS) active-pixel sensor a camera, video camera, or other devices.Image sensor 202 generates raw image data that is sent to SOC component204 for further processing. In some embodiments, the image dataprocessed by SOC component 204 is displayed on display 216, stored insystem memory 230, persistent storage 228 or sent to a remote computingdevice via network connection. The raw image data generated by imagesensor 202 may be in a Bayer color kernel array (CFA) pattern(hereinafter also referred to as “Bayer pattern”).

Motion sensor 234 is a component or a set of components for sensingmotion of device 100. Motion sensor 234 may generate sensor signalsindicative of orientation and/or acceleration of device 100. The sensorsignals are sent to SOC component 204 for various operations such asturning on device 100 or rotating images displayed on display 216.

Display 216 is a component for displaying images as generated by SOCcomponent 204. Display 216 may include, for example, liquid crystaldisplay (LCD) device or an organic light emitting diode (OLED) device.Based on data received from SOC component 204, display 216 may displayvarious images, such as menus, selected operating parameters, imagescaptured by image sensor 202 and processed by SOC component 204, and/orother information received from a user interface of device 100 (notshown).

System memory 230 is a component for storing instructions for executionby SOC component 204 and for storing data processed by SOC component204. System memory 230 may be embodied as any type of memory including,for example, dynamic random access memory (DRAM), synchronous DRAM(SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM),static RAM (SRAM) or a combination thereof. In some embodiments, systemmemory 230 may store pixel data or other image data or statistics invarious formats.

Persistent storage 228 is a component for storing data in a non-volatilemanner. Persistent storage 228 retains data even when power is notavailable. Persistent storage 228 may be embodied as read-only memory(ROM), flash memory or other non-volatile random access memory devices.

SOC component 204 is embodied as one or more integrated circuit (IC)chip and performs various data processing processes. SOC component 204may include, among other subcomponents, image signal processor (ISP)206, a central processor unit (CPU) 208, a network interface 210, sensorinterface 212, display controller 214, neural processor circuit 218,graphics processor (GPU) 220, memory controller 222, video encoder 224,storage controller 226, and bus 232 connecting these subcomponents. SOCcomponent 204 may include more or fewer subcomponents than those shownin FIG. 2 .

ISP 206 is hardware that performs various stages of an image processingpipeline. In some embodiments, ISP 206 may receive raw image data fromimage sensor 202, and process the raw image data into a form that isusable by other subcomponents of SOC component 204 or components ofdevice 100. ISP 206 may perform various image-manipulation operationssuch as image translation operations, horizontal and vertical scaling,color space conversion and/or image stabilization transformations, asdescribed below in detail with reference to FIG. 3 .

CPU 208 may be embodied using any suitable instruction set architecture,and may be configured to execute instructions defined in thatinstruction set architecture. CPU 208 may be general-purpose or embeddedprocessors using any of a variety of instruction set architectures(ISAs), such as the x86, PowerPC, SPARC, RISC, ARM or MIPS ISAs, or anyother suitable ISA. Although a single CPU is illustrated in FIG. 2 , SOCcomponent 204 may include multiple CPUs. In multiprocessor systems, eachof the CPUs may commonly, but not necessarily, implement the same ISA.

Graphics processing unit (GPU) 220 is graphics processing circuitry forperforming graphical data. For example, GPU 220 may render objects to bedisplayed into a frame buffer (e.g., one that includes pixel data for anentire frame). GPU 220 may include one or more graphics processors thatmay execute graphics software to perform a part or all of the graphicsoperation, or hardware acceleration of certain graphics operations.

Neural processor circuit 218 is a circuit that performs various machinelearning operations based on computations including multiplication,addition and accumulation. Such computations may be arranged to perform,for example, convolution of input data and kernel data. Neural processorcircuit 218 is a configurable circuit that performs these operations ina fast and power-efficient manner while relieving CPU 208 ofresource-intensive operations associated with neural network operations.Neural processor circuit 218 may receive the input data from sensorinterface 302, the image signal processor 206, system memory 230 orother sources such as network interface 210 or GPU 220. The output ofneural processor circuit 218 may be provided to various components ofdevice 100 such as the image signal processor 206, system memory 230 orCPU 208 for various operations. The structure and operation of neuralprocessor circuit 218 is described below in detail with reference toFIG. 3 .

Network interface 210 is a subcomponent that enables data to beexchanged between devices 100 and other devices via one or more networks(e.g., carrier or agent devices). For example, video or other image datamay be received from other devices via network interface 210 and bestored in system memory 230 for subsequent processing (e.g., via aback-end interface to image signal processor 206, such as discussedbelow in FIG. 3 ) and display. The networks may include, but are notlimited to, Local Area Networks (LANs) (e.g., an Ethernet or corporatenetwork) and Wide Area Networks (WANs). The image data received vianetwork interface 210 may undergo image processing processes by ISP 206.

Sensor interface 212 is circuitry for interfacing with motion sensor234. Sensor interface 212 receives sensor information from motion sensor234 and processes the sensor information to determine the orientation ormovement of the device 100.

Display controller 214 is circuitry for sending image data to bedisplayed on display 216. Display controller 214 receives the image datafrom ISP 206, CPU 208, graphic processor or system memory 230 andprocesses the image data into a format suitable for display on display216.

Memory controller 222 is circuitry for communicating with system memory230. Memory controller 222 may read data from system memory 230 forprocessing by ISP 206, CPU 208, GPU 220 or other subcomponents of SOCcomponent 204. Memory controller 222 may also write data to systemmemory 230 received from various subcomponents of SOC component 204.

Video encoder 224 is hardware, software, firmware or a combinationthereof for encoding video data into a format suitable for storing inpersistent storage 128 or for passing the data to network interface 210for transmission over a network to another device.

In some embodiments, one or more subcomponents of SOC component 204 orsome functionality of these subcomponents may be performed by softwarecomponents executed on ISP 206, CPU 208 or GPU 220. Such softwarecomponents may be stored in system memory 230, persistent storage 228 oranother device communicating with device 100 via network interface 210.

Image data or video data may flow through various data paths within SOCcomponent 204. In one example, raw image data may be generated from theimage sensor 202 and processed by ISP 206, and then sent to systemmemory 230 via bus 232 and memory controller 222. After the image datais stored in system memory 230, it may be accessed by video encoder 224for encoding or by display 116 for displaying via bus 232.

Example Neural Processor Circuit

Neural processor circuit 218 is a configurable circuit that performsneural network operations on the input data based at least on kerneldata 340. For this purpose, neural processor circuit 218 may include,among other components, neural task manager 310, a plurality of neuralengines 314A through 314N (hereinafter collectively referred as “neuralengines 314” and individually also referred to as “neural engine 314”),kernel direct memory access (DMA) 324, data buffer 318 and buffer DMA320. Neural processor circuit 218 may include other components notillustrated in FIG. 3 .

Each of neural engines 314 performs computing operations for neuralnetwork operations in parallel. Depending on the load of operation,entire set of neural engines 314 may be operated or only a subset of theneural engines 314 may be operated while the remaining neural engines314 are placed in a power save mode to conserve power. Each of neuralengines 314 includes components for storing one or more kernels, forperforming multiply-accumulate operations, and for post-processing togenerate an output data 328, as described below in detail with referenceto FIG. 4 . One example of a neural network operation is a convolutionoperation.

Neural task manager 310 manages the overall operation of neuralprocessor circuit 218. Neural task manager 310 may receive a task listfrom a compiler executed by CPU 208, store tasks in its task queues,choose a task to perform, and send instructions to other components ofthe neural processor circuit 218 for performing the chosen task. Neuraltask manager 310 may also perform switching of tasks on detection ofevents such as receiving instructions from CPU 208. In one or moreembodiments, the neural task manager 310 sends rasterizer information tothe components of the neural processor circuit 218 to enable each of thecomponents to track, retrieve or process appropriate portions of theinput data and kernel data, as described below in detail with referenceto FIGS. 5 through 7 . Although neural task manager 310 is illustratedin FIG. 3 as part of neural processor circuit 218, neural task manager310 may be a component outside the neural processor circuit 218.

Kernel DMA 324 is a read circuit that fetches kernel data from a source(e.g., system memory 230) and sends kernel data 326A through 326N toeach of the neural engines 314. Kernel data represents information fromwhich kernel elements can be extracted. In one embodiment, the kerneldata may be in a compressed format which is decompressed at each ofneural engines 314. Although kernel data provided to each of neuralengines 314 may be the same in some instances, the kernel data providedto each of neural engines 314 is different in most instances.

Data buffer 318 is a temporary storage for storing data associated withthe neural network operations. In one embodiment, data buffer 318 isembodied as a memory that can be accessed by all of the neural engines314. Data buffer 318 may store input data 322A through 322N for feedingto corresponding neural engines 314A through 314N, as well as outputfrom each of neural engines 314A through 314N for feeding back intoneural engines 314 or sending to a target circuit (e.g., system memory230). The operations of data buffer 318 and other components of theneural processor circuit 218 are coordinated so that the input data andintermediate data stored in the data buffer 318 is reused acrossmultiple operations at the neural engines 314, and thereby reduce datatransfer to and from system memory 230. Data buffer 318 may be operatedin a broadcast mode where input data of all input channels are fed toall neural engines 314 or in a unicast mode where input data of a subsetof input channels are fed to each neural engine 314.

The input data 322 stored in data buffer 318 can be part of, amongothers, image data, histogram of oriented gradients (HOG) data, audiodata, meta data, output data 328 of a previous cycle of the neuralengine 314, and other processed data received from other components ofthe SOC component 204.

Buffer DMA 320 includes a read circuit that receives a portion (e.g.,tile) of the input data from a source (e.g., system memory 230) forstoring in data buffer 318, and a write circuit that forwards data fromdata buffer 318 to a target (e.g., system memory).

Example Neural Engine Architecture

FIG. 4 is a block diagram of the neural engine 314, according to oneembodiment. The neural engine 314 performs various operations tofacilitate neural network operations such as convolution, spatialpooling and local response normalization. The neural engine 314 receivesthe input data 322, performs multiply-accumulate operations (e.g.,convolution operations) on the input data 322 based on stored kerneldata, performs further post-processing operations on the result of themultiply-accumulate operations, and generates the output data 328. Theinput data 322 and/or the output data 328 of the neural engine 314 maybe of a single channel or multiple channels.

Neural engine 314 may include, among other components, input buffercircuit 402, computation core 416, neural engine (NE) control 418,kernel extract circuit 432, accumulators 414 and output circuit 424.Neural engine 314 may include further components not illustrated in FIG.4 .

Input buffer circuit 402 is a circuit that stores a portion of the inputdata 322 as it is received from the data buffer 318 and sends anappropriate portion 408 of input data for a current task or process loopto computation core 416 for processing. Input buffer circuit 402includes a shifter 410 that shifts read locations of input buffercircuit 402 to change the portion 408 of input data sent to computationcore 416. By changing portions of input data provided to the computationcore 416 via shifting, neural engine 314 can perform multiply-accumulatefor different portions of input data based on fewer number of readoperations. In one or more embodiments, the input data 322 includes dataof different convolution groups and/or input channels.

Kernel extract circuit 432 is a circuit that receives kernel data 326from kernel DMA 324 and extracts kernel coefficients 422. In oneembodiment, the kernel extract circuit 432 references a look up table(LUT) and uses a mask to reconstruct a kernel from compressed kerneldata 326. The mask indicates locations in the reconstructed kernel to bepadded with zero and remaining locations to be filled with numbers. Thekernel coefficients 422 of the reconstructed kernel are sent tocomputation core 416 to populate register in multiply-add (MAD) circuitsof computation core 416. In other embodiments, the kernel extractcircuit 432 receives kernel data in an uncompressed format and thekernel coefficients are determined without referencing a LUT or using amask.

Computation core 416 is a programmable circuit that performs computationoperations. For this purpose, the computation core 416 may include MADcircuits MAD0 through MADN and a post-processor 428. Each of MADcircuits MAD0 through MADN may store an input value in the portion 408of the input data and a corresponding kernel coefficient in the kernelcoefficients 422. The input value and the corresponding kernelcoefficient are multiplied in each of MAD circuits to generate aprocessed value 412.

Accumulator 414 is a memory circuit that receives and stores processedvalues 412 from MAD circuits. The processed values stored in accumulator414 may be sent back as feedback information 419 for further multiplyand add operations at MAD circuits or sent to post-processor 428 forpost-processing. Accumulator 414 in combination with MAD circuits form amultiply-accumulator (MAC) 404. In one or more embodiments, accumulator414 may have subunits where each subunit sends data to differentcomponents of neural engine 314. For example, during a processing cycle,data stored in a first subunit of accumulator 414 is sent to MAC circuitwhile data stored in a second subunit of accumulator 414 is sent topost-processor 428.

Post-processor 428 is a circuit that performs further processing ofvalues 412 received from accumulator 414. The post-processor 428 mayperform operations including, but not limited to, applying nonlinearfunctions (e.g., Rectified Linear Unit (ReLU)), normalizedcross-correlation (NCC), merging the results of performing neuraloperations on 8-bit data into 16-bit data, and local responsenormalization (LRN). The result of such operations is output from thepost-processor 428 as processed values 417 to output circuit 424.

NE control 418 controls operations of other components of the neuralengine 314 based on the operation modes and parameters of neuralprocessor circuit 218. Depending on different modes of operation (e.g.,group convolution mode or non-group convolution mode) or parameters(e.g., the number of input channels and the number of output channels),neural engine 314 may operate on different input data in differentsequences, return different values from accumulator 414 to MAD circuits,and perform different types of post-processing operations at postprocessor 428. To configure components of the neural engine 314 tooperate in a desired manner, the NE control 418 sends a control signalto components of the neural engine. NE control 418 may also includerasterizer 430 that tracks the current task or process loop beingprocessed at neural engine 314, as described below in detail withreference to FIG. 5 through 7 .

Output circuit 424 receives processed values 417 from the post-processor428 and interfaces with data buffer 318 to store processed values 417 indata buffer 318. For this purpose, output circuit 424 may send out asoutput data 328 in a sequence or a format that is different from thesequence or format in which the processed values 417 are processed inpost-processor 428.

The components in the neural engine 314 may be configured during aconfiguration period by the NE control 418 and the neural task manager310. For this purpose, the neural task manager 310 sends configurationinformation to the neural engine 314 during the configuration period.The configurable parameters and modes may include, but are not limitedto, mapping between input data elements and kernel elements, the numberof input channels, the number of output channels, performing of outputstrides, and enabling/selection of post-processing operations at thepost processor 428.

Operation of Segmenting of Data for Processing at Neural ProcessorCircuit

Input data is typically split into smaller pieces of data for parallelprocessing at multiple neural engines 314. Often multiple cycles ofoperations are performed to generate output for a task associated with aneural network. A compiler executed by CPU 208 analyzes the hierarchyand nodes of the neural network and determines how the input data is tobe segmented based on the hardware constraints of the neural processorcircuit 218. One of functions of the compiler is to determine how inputdata is to be split into smaller data units for processing at the neuralengines 314, and how the processing is to be iterated in loops toproduce the result for tasks.

FIG. 5 is a conceptual diagram illustrating loops for processing theinput data at neural processor circuit 218, according to one embodiment.The outermost loop represents processing for a convolution group, ifgroup convolution involving multiple convolution group is used. Groupconvolutions are convolutions where input data of the input channels ineach group are used only for generating output data of output channelsof each group but are not used for generating output data for outputchannels of other groups. Hence, each group of the group convolution canbe treated as a separate convolution operation.

In the loop for each convolution group is a processing loop for a sliceof the input data. The entire input data for a convolution operation issegmented into multiple strips of slices in an overlapping manner, asshown in FIG. 6 . The overlapping portions 602, 604, 606 are parts ofthe input data that are overfetched in two adjacent slices to providespatial support for a corresponding kernel. The second outermost loopperforms convolution operation for each slice in the input data. Withinthe loop for a slice is a processing loop for a tile of the slice. Eachslice is segmented into a plurality of tiles, as shown in FIG. 6 . Theoverlapping portions 608, 610, 612, 614 are parts of the input data inslice 4 that are overfetched in two adjacent tiles to provide spatialsupport for a corresponding kernel. The rightmost tile will typicallyhave a width smaller than other tiles of the slice. In one embodiment,input data for each tile is loaded onto data buffer 318 in a read cycleand reused for operations in processing loops for the tile. In theprocessing loop for the tile is a processing loop for a work unit. Eachtile is segmented into multiple work units as shown in FIG. 6 . A workunit is a portion of the input data having a size that produces outputvalues that fit into accumulator 414 of neural engine 314 during asingle cycle of the computation core 416. Although the shape of eachwork unit is shown as a horizontal strip in FIG. 6 , the shape of thework unit can be different depending on the shape and size of the tile.The work units also have overlapping parts that represent overfetcheddata to provide support for a corresponding kernel. Especially, workunits for the last tile of a slice may have a shape of a vertical stripif the tile is tall. In one or more embodiments, the size of each workunit is 256 bytes. In such embodiments, for example, work units can beshaped to one of 16×16, 32×8, 64×4, 128×2 or 256×1 dimension.

For each work unit, an internal processing loop may be provided for anoutput channel group (OCG). The number of output channels produced for agiven work unit by a single cycle of the computation core 416 isreferred to as an OCG. Depending on operation modes, each neural engine314 may process output data of different numbers of output channels(e.g., 8 channels, 32 channels) for a single load of input data into itsinput buffer circuit 402.

For each output channel group, an internal processing loop may beprovided for an input channel (Cin). If an input stride is implementedto skip certain input data, loops for sub-input channels (Sub-Cin) maybe provided within the processing loop for the input channel (Cin).

For each input channel or each sub-input channel, internal loops areprovided for processing horizontal spatial support for a kernel and thevertical support within each horizontal spatial support. The spatialsupport refers to the input data for convolution with the kernel, andincludes overfetched input data for performing convolution at the edgesof the input data.

Overfetch refers to fetching additional input data in current slice,tile or work unit so that proper dimension of input data can be providedfor convolution with a kernel. In one or more embodiments, overfetch isperformed vertically between slices to obtain additional rows of inputdata (shown as overlapping portions 602, 604, 606 in FIG. 6 ),horizontally between tiles to obtain additional columns of input data(shown as overlapping portions 608, 610, 612, 614 in FIG. 6 ), andvertically between work units within a tile to obtain additional rows ofinput data.

For each spatial support for the kernel, an internal processing loop foran output channel (OC) is provided to generate output data for eachoutput channel (Cout). In cases where output stride implements a spatialupsampling, an additional inner loop for processing each sub-outputchannel is provided. Loading of kernel coefficients and MAC operationsare performed within the loop for the output channel (OC) or sub-outputchannel if an output stride is implemented, to generate output data forthe output channel (OC) or sub-output channel.

The nested loop structure of FIG. 5 is merely illustrative. Loops may beomitted, added or structured differently depending on various factors.For example, if only a single convolution group is used, the outermostloop may be removed. Further, the loop structure for the horizontalspatial support and the vertical spatial support may be reversed.

In one or more embodiments, the operations associated dividing the inputspace into smaller units and processing these smaller units as describedabove with reference to FIGS. 5 and 6 are performed by rasterizers 714,718, 720, 722 in various components of neural processor circuit 218. Arasterizer is a circuit in various components of neural processorcircuit 218 that keeps track of the segment of the input/output data(e.g., group, work unit, input channel, output channel) and instructsthe components of neural processor circuit for proper handling of thesegment of the input data. For example, rasterizer 720 in buffer DMA 320tracks tiles and slices received from system memory 230 while rasterizer718 in data buffer 318 broadcasts in sequence work units for processingby the neural engines 314. Rasterizer 724 in kernel DMA 322 determineswhich kernels are to be received and distributed to neural engines 314,while rasterizers 714 in neural engines 314 operate shifters 410 ininput buffer circuits 402 to forward correct portions 408 of input datato MAC 404, and send the finished output data 328 to the data buffer318.

FIG. 7 is a diagram illustrating programming of rasterizers 714, 718,720, 722 in components 314, 318, 320, 322 of the neural processorcircuit 218, according to one embodiment. To perform their functions,each of rasterizers 714, 718, 720, 722 receives task information 710indicating how the input data and/or kernel data are to be segmented andto be handled by each component of the neural processor circuit 218. Thetask information includes information about particulars of the currentlayer (e.g., dimensions of input and output data, dimension of anassociated kernel, types of padding at the boundaries of input data).Rasterizers 714, 718, 720, 722 may also receive constraints on theiroperations (e.g., whether to allow or disallow tile width over athreshold).

By providing rasterizers in different components of neural processorcircuit 218, overhead in data transmitted between the components of theneural processor circuit 218 may be reduced. If a single centralrasterizer is provided to control different components of the neuralprocessor circuit 218, kernel data, input data, and output datatransmitted between the components may be needed in these data toidentify associated position in the loops of the task such asconvolution group, tile, slice, work unit, input channel and outputchannel. By using distributed rasterizers, no separate metadata isneeded to transmit the kernel data, input data and output data amongcomponents of the neural processor circuit 218.

Example Process at Neural Engine Architecture

FIG. 8 is a flowchart illustrating a method of processing input data inneural processor circuit 218, according to one embodiment. After neuraltask manager 310 programs rasterizers 714, 718, 720, 722, the process ofoperating buffer DMA 320 is initiated by rasterizer 720 instructing 804buffer DMA 320 to cause buffer DMA 320 to receive a tile of input datafrom system memory 230. The tile received by buffer DMA 320 is stored806 in data buffer 318.

Rasterizer 718 in data buffer 318 then instructs 808 data buffer 318 tosend a work unit to one or more neural engines 314. The work unit isthen stored in input buffer circuits 402 of the one or more neuralengines 314.

In one or more embodiments, input buffer circuit 402 selects 816 aportion of work unit to be sent to MAC 404 to performmultiply-accumulate operation. Then MAC 404 performs 820multiply-accumulate operations on the selected portion of the work unitusing a corresponding kernel. Then it is determined 824 if the entirework unit is processed at one or more neural engines 314. If not, theselected portion of the work unit is shifted 828 by shifter 410 andreturns to perform 820 another round of multiply-accumulate operations.

If it is determined 824 that the entire work unit was processed, then itproceeds to determine 832 if all work units in the tile was processed.If not, then the process proceeds 836 to the next work unit by havingdata buffer 318 send 808 a next work unit to one or more neural engines314, and repeats the subsequent processes.

If it is determined 832 that all work units in the tile was processed bythe neural engines 314, the process proceeds to determine 840 whetherall tiles for the input data were processed. If not, the processproceeds 844 to a next tile by having rasterizer 720 instructs 804buffer DMA 320 to receive a next tile from system memory 230 and repeatsthe subsequent processes.

If it is determined 840 that all tiles of the input data are processed,then the process ends for the current input data. Then, the process maybe repeated to process the next input data or proceed to the next task.

Embodiments of the process as described above with reference to FIG. 8are merely illustrative. Further loops may be embodied, as describedabove with reference to FIG. 5 . Moreover, sequence of the process maybe modified or omitted.

Example Multiply-Accumulator Circuit

FIGS. 9A and 9B illustrate an example multiply-accumulator (MAC) 404that supports neural networking operations using fixed-point andfloating-point operands operating in a fixed-point mode of operation,according to one embodiment. Each neural engine 314 of a neuralprocessor circuit 218 may include the MAC 404. The MAC 404 includes aMAD 918 and an accumulator 414. Although a single MAD 918 is shown beingcoupled to the accumulator 414, each MAC 404 of a neural engine 314 mayinclude multiple (e.g., 256) MADs that are coupled to the accumulator414 as shown for the MAD 918. Each MAD 918 includes a multiplier 906, ashift register 910, an adder 914, a shift offset register 922, and anexponent adder 926. The multiplier 906 is coupled to the shift register910. The exponent adder 926 is coupled to the shift offset register 922,and the shift register 910. The shift register 910 is coupled to anadder 914, which is coupled to an accumulator 414.

The neural processor circuit 218 may include multiple neural engines 314where each neural engine 314 includes a MAC 404 with multiple MADcircuits coupled to an accumulator 414. For the sake of convenience, theoperation of a single MAD 918, instead of multiple MADs, is discussedherein. Multiple MADs 918 may operate in parallel using differentoperands to execute a neural network operation.

The MAC 404 may operate in a floating-point mode of operation or afixed-point mode of operation. In the fixed-point mode of operation, theMAC 404 receives fixed-point (e.g., INT8) input data 900 for convolutionwith a fixed-point kernel coefficient 422. In a floating-point mode ofoperation, the MAC 404 receives 16-bit floating-point (e.g., FP16) inputdata 900 for convolution with a floating-point kernel coefficient 422.In some embodiments, the neural processor circuit 218 includes 8 neuralengines, each neural engine 314 including 256 MADs. In the fixed-pointmode, the neural processor circuit 218 receives as input data 256 byteswhich is treated as 256 8-bit integers, multiplies the 256 8-bitintegers by a single kernel coefficient, and produces 256partial-product results (which is accumulate into one of the256-component accumulators), eventually producing a single 256-componentresult, which is typically emitted as a 256-byte result (256 8-bitintegers). For a neural processor circuit 218 including 8 neural engines314 and each neural engine 314 including 256 MADs, the floating-pointmode of operation supports 128 multiply-add operations in parallel foreach neural engine 314 while processing a 256-byte work unit in aprocessing cycle across the 8 neural engines. In the floating-pointmode, the neural processor circuit 218 receives as input data 256 byteswhich is treat as 128 16-bit floating-point numbers (FP16). The neuralprocessor circuit 218 also receives two kernel coefficients, andmultiplies the 128 input floats by both kernel coefficients, producingtwo 128-component results. The resulting 256 component partial-product(2×128) are accumulated into a single 256-component accumulator, andeventually produce two 128-component outputs. This is typically emittedas a pair of 128-component FP16 channels.

The MAC 404 may include 256 multiply-add (MAD) 918 circuits to processwork units. Each MAD 918 uses an accumulator 414 for multi-processingcycle multiply-add operations within the MAC 404. In some embodiments,the MAC 404 includes a 32-bit accumulator used by the MAC 404 forstoring the output data of MADs as accumulated values 930 from one ormore processing cycles. Each 32-bit entry in the accumulator 414 may beused as an accumulated value 930 for an addition operation with themultiplied value 908/912 of a subsequent (e.g., next) processing cycle.The accumulator 414 selectively provides the output data to the MAD 918as an accumulated value 930, or the post-processor 428 when accumulationof multiplied values from multiple processing cycles is complete.

Example Fixed-Point Convolution

FIG. 9A illustrates the shift register 922, exponent adder 926, andshift register 910 are deactivated in the fixed-point mode of operation,according to one embodiment. The multiplier 906 is coupled to the inputof the MAD 918 to receive input data 900 and a kernel coefficient 422,and multiplies the input data 900 and the kernel coefficient 422 togenerate a multiplied value 908.

The multiplier 906 is coupled to the shift register 910 that bypassesthe multiplied value 908 to the adder 914. The adder 914 adds themultiplied value 908 of the current processing with an accumulated value930 from the accumulator 414 from one or more prior processing cycles togenerate output data 942. The accumulated value 930 that is provided tothe adder 914 may include an output from the MAD 918. If there is noaccumulated value 930 to add with the multiplied value 908, themultiplied value 908 is stored in the accumulator 414 as the output data942. The stored output data 942 may be provided as an input accumulatedvalue 930 to an adder 914 of a MAD 918 for a subsequent processingcycle.

The MAC 404 may execute multiple (e.g., 256) multiply-accumulateoperations per processing cycle using input data 900 and kernelcoefficients 422 having an INT8 precision. Each of neural engines 314may process multiple (e.g., 8) output channels (e.g., one output channelper accumulator 414). In some embodiments, the accumulators 414 can acteither as a set of eight 256-component accumulators (which allows theneural processor circuit 218 to compute eight output channels at a timeper pass through the input data), or they can be divided into twofour-accumulator pools. These are used as a “double buffer”—the MAC 404portion of the neural engine 314 computes four output channels while thepost-processor 428 consumes the previously-computed output (the previousfour computed output channels). Operating in this “ping pong” modeallows the post-process to overlap with the convolution taking place inthe MAC 404, but at a cost of (potentially) requiring twice as manypasses through the input data (because half as many output channels areproduced per pass).

In some embodiments, following the fixed-point multiplication, the MAD918 provides the fixed-point multiplied value 908 to the adder 914 to beadded to the accumulated value 930 stored in the accumulator 414 fromone or more previous processing cycles. Because the fixed-pointmultiplied value 908 does not require realignment, the shift register910, exponent adder 926, and shift offset register 922 can be turned offor operated in a low power mode during the fixed-point mode of operationfor power conservation, and are bypassed before reaching the adder 914,as indicated by the dotted lines in FIG. 9A. In some embodiments, thefixed-point multiplied value 908 is sign-extended to 32-bits and addedto fixed-point 32-bit accumulated value 930 from the accumulator 414.The fixed-point 32-bit sum, or output data 942, is then stored in theaccumulator to be used as an accumulated value 930 in a subsequentprocessing cycle.

Example Floating-Point Convolution

FIG. 9B illustrates the shift register 922, exponent adder 926, andshift register 910 activated in the floating-point mode of operation,according to one embodiment. For the floating-point mode of operation,the input data is separated into an input data mantissa 938 and an inputdata exponent 936.

The kernel coefficient is separated into a kernel coefficient mantissa932, and a kernel coefficient exponent 920. The multiplier 906 receivesthe input data mantissa 938 and the kernel coefficient mantissa 932, andmultiplies these values to generate a mantissa value 948. The exponentadder 926 adds the input data exponent 936 and the kernel coefficientexponent 920 to generate an exponent value, and the exponent adder 926modifies the exponent value using an offset value from the shift offsetregister 922.

The shift register 910 generates the multiplied value by realigning themantissa value 948 based on the exponent value 928 to generate amultiplied value 912. The adder 914 adds the multiplied value 912 withan accumulator value 930 from the accumulator 414 from one or more priorprocessing cycles. The accumulated value 930 that is provided to theadder 914 may include an output from the MAD 918. If there is noaccumulated value 930 to add with the multiplied value 912, themultiplied value 912 is stored in the accumulator 414. The storedmultiplied value 912 may be provided as an input accumulated value 930to an adder 914 of a MAD 918 for a subsequent processing cycle.

In some embodiments, the shift register 910 uses binary sum to determinea shifting amount for aligning the binary point 924 of the multipliedvalue 912 for fixed-point addition in the adder 914 (e.g., convertingthe floating-point multiplied value 908 into a fixed-point integer). Theshift register 910 uses an arithmetic shift to align the binary point924 and extend bit size of the multiplied value 908 (e.g., extend a23-bit multiplied value 908 to 32-bits) that corresponds to the bit sizeof the accumulator 414.

In some embodiments, the shift offset register 922 contains a 5-bitvalue indicating a binary point 924 position to be used in part by theshift register 910 for aligning a floating-point multiplied value to afixed point precision of the accumulator 414.

In some embodiments, the exponent adder 926 takes input data exponent936, kernel coefficient exponent 920, and binary point 924 as input toproduce an exponent value (or binary sum 928) which is a 5-bit binarysum. The binary sum is used to align a floating-point multiplied value908 for fixed-point addition by the adder 914.

In some embodiments, the floating-point precision may include processingoperands having more bits than those used in the fixed-point mode ofoperation. For example, because FP16 includes twice as many bits asINT8, a work unit of 256-bytes may require a different MAD arrangementthan the arrangement used for fixed-point mode. In this example, the256-byte work unit would utilize only 128 MADs 918 in the MAC 404 ratherthan 256, decreasing the output bandwidth to half of that utilized inthe fixed-point mode of operation. In order to compensate for apotential loss of bandwidth, a first portion of MAD 918 circuits can beused for processing multiply-accumulate operations using the work unitand a first kernel coefficient 422, and a second portion of MAD 918circuits can be used from processing the same work unit using a secondkernel coefficient. Whereas in the fixed-point mode of operation onekernel coefficient 422 is processed per clock cycle by the MAC 404,using a second kernel coefficient for processing the same work unit inparallel with a first kernel coefficient affords the MAC 404 circuit thefull use of all 256 MADs 918. In some embodiments, 128multiply-accumulate operations are designated for an even output channeland 128 multiply-accumulate operations are designated for an odd outputchannel. In total, the neural engine 314 can generate two outputchannels per each processing cycle of the neural engine 314 in thefloating-point mode of operation.

In some embodiments, the MAC 404 receives input data 900 and kernelcoefficients 422 in the floating-point mode of operation similarly toreceiving input data 900 and kernel coefficients 422 in the fixed-pointmode of operation. For example, the input buffer circuit 402 broadcasts256-bytes of input data 900 portions distributed across 256 MAD 918circuits, where each MAD 918 is mapped to a portion of input data (e.g.,pixel 0 sent to MAD0, pixel 1 sent to MAD1, and so on). However, inorder to make use of all 256 MAD 918 circuits, two separate kernelcoefficients 422 are processed with the same pixel data 900 from thework unit. In some embodiments, kernel coefficients with a value of 0can be skipped, thus taking advantage of kernel sparsity in order toconserve power and optimize each processing cycle.

In some embodiments, each floating point operand can be represented ass*m*2^(e), where s represents the sign bit, m represents the mantissa,and e represents the exponent. Rather than introducing separatecomponents to handle a floating-point mode or a fixed-point mode ofoperation, the MAD 918 separates each floating point input into itsconstituent parts, separating the sign bit and mantissa bits from theexponent bits. This enables the MAD 918 to process floating-point (e.g.,FP16) precision using the shared circuitry that is used for processingfixed-point (e.g., INT8) by activating the exponent adder 926, shiftregister 910, and shift offset register 922 in the floating-point modeof operation.

FIG. 10 illustrates a process for determining a binary point positionand aligning a multiplied value 908, according to one embodiment. Asshown, the input data mantissa 938 and kernel coefficient mantissa 932have a precision of 1.10 (i.e., one bit for MSB integer representationand 10 bits for mantissa) and values ranging from 1 to 2 (i.e.,1.0000000000 to 1.1111111111˜2). The input data mantissa 938 and kernelcoefficient mantissa 932 are multiplied by the multiplier 906, resultingin a 23-bit multiplied value 912 with a precision of 2.20 (i.e., twobits for MSB integer representation and 20 bits for mantissa) and valuesranging from 1 to 4 (i.e., 01.00000000000000000000 to11.11111111000000000001˜4).

To define the magnitude of the 23-bit multiplied value 912, theexponents of both inputs are added with an additional binary point 924value in the exponent adder 926. Input data exponent 936 is added tokernel coefficient exponent 920 in addition to a 5-bit binary point 924value in the shift offset register 922. The binary point 924 is aconfigurable global offset value determined by the compiler during acompilation operation, and may be set based in part on kernel size. Forexample, a 4×4 kernel used for convolution involves more multiply-addoperations and generates more partial products than a 2×2 kernel. Inthis case, the compiler may assign a lower binary point 924 value inorder to accommodate a larger range for multiply-accumulate operationsto reduce the likelihood of overflow. Conversely, for smaller kernelsizes (e.g., the 2×2 kernel), fewer multiply-accumulate operations areperformed due to having fewer kernel coefficients. In this case, thecompiler may designate a larger binary point 924 value to afford alarger precision with a smaller range. In one or more embodiments, thesetrade-offs are determined by a trained model during compilation. Theexponent adder 926 adds the binary point 924 to the input data exponent936 and the kernel coefficient exponent 920 to generate a 5-bit binarysum 928.

The binary sum 928 value is used to drive the shift register 910 inorder to align the multiplied value 912 for addition with the (e.g.,32-bit) fixed-point accumulated value 930 in the accumulator 414. Asillustrated in FIG. 10 , the 23-bit multiplied value 912 is shiftedaccording to the amount of precision and/or range needed to supportaddition and accumulation operations. The shift register 910 performsarithmetic shifts on the 23-bit multiplied value 912 to maintain itssign while aligning the 23-bit multiplied value 912 using the binarypoint 924. The most-significant bits (MSB) are sign extended on theleft, and remaining bits are padded with zeros on the right, producing afixed-point 32-bit multiplied value 912 comprised of a sign bit, Minteger bits 1020, and N fractional bits 1022. The accumulators may usetwo's complement representations or signed-magnitude representations.

The fixed-point 32-bit multiplied value 912 is sent to the adder 914 foraddition operations with either a value of 0 or a configurable biasvalue if operating in a first processing cycle, or fixed-point 32-bitaccumulated value 930 from previous processing cycles if operating in asubsequent processing cycle. In some embodiments, fixed-pointrepresentations of floating-point values in the accumulator 414 areconverted back to floating-point values (e.g., FP16) in post-processing.

Binary Point Configuration for Mixed Precision Convolution

The MAC 404 may support multiply-accumulate operations using input data900 and kernel coefficients 422 of different precisions. The componentsof each MAD 918 enable the MAC 404 to process two operands havingrespective fixed-point (e.g., INT8) and floating-point (e.g., FP16)precisions in the same multiply-add operations, and provide for theproper alignment needed for accumulation. A predetermined binary point924 position specifies the amount of shift applied by the shift register910 based on the types of input received. For example, if two inputs arereceived having INT8 precision, the binary point location is 0. If twoinputs are received having an INT8 precision and a FP16 precision,respectively, the binary point location is 10. Lastly, if two inputs arereceived having FP16 precision, the binary point location is 20.

For example, if FP16 input data 900 is received with an INT8 kernelcoefficient 422, the MAD 918 can separate the FP16 input data 900 intoits contingent parts, forming an input data mantissa 938 and an inputdata exponent 936 as discussed above with reference to FIG. 9B. However,the MAD 918 also effectively separates the INT8 kernel coefficient byassigning predetermined binary point 924 value to be processed in theaddition operation of the exponent adder 926. The input data exponent936 is added to a kernel coefficient exponent 920 value of 0 and thebinary point 924 value of 10 (e.g., predetermined binary point 924 valuefor INT8 and FP16 operands). In addition, the input data mantissa 938 ismultiplied by the INT8 kernel coefficient 422 in the multiplier 906. Themultiplied value 908 is then shifted accordingly and added to theprocessed results in the accumulator 414. Similarly, the MAD 918 mayreceive INT8 input data 900 and a FP16 kernel coefficient producing thesame result.

Example Processes at MAD Circuit

FIG. 11 is a flowchart illustrating a method of performingmultiply-accumulation operations on input data and kernel coefficientshaving either fixed-point precision or floating-point precision,according to one embodiment.

First, the MAD circuit multiplies 1100 input data and kernel data togenerate a multiplied value of a processing cycle. Then, the MAD circuitadds 1102 the multiplied value to a first accumulated value from aprevious processing cycle to generate output data. The input data andkernel coefficients may be fixed point or floating point values. Iffloating point values are used, the multiplied value may be converted toa fixed point precision of the accumulated value stored in theaccumulator.

The accumulator stores 1104 the output data obtained as a result of theaddition. Then, the accumulator provides 1106 the output data to the MADcircuit as an accumulated value for subsequent processing cycles.

FIG. 12 illustrates a method of processing fixed-point input data by akernel coefficient, according to one embodiment. The MAC receives 1200fixed-point input data and a fixed-point kernel coefficient. The shiftoffset register, exponent adder, and shift register of the MAC isdeactivated for the fixed-point mode of operation.

The multiplier multiplies 1204 the input data and kernel coefficient togenerate a multiplied value. Then, the adder adds 1206 the multipliedvalue with the accumulated value from the accumulator to generate outputdata.

The output data is stored 1208 in accumulator. The accumulator provides1210 output data to MAD as accumulated value for another processingcycle.

FIG. 13 illustrates an example method of processing floating-point inputdata by kernel coefficients, according to one embodiment. The MADreceives 1300 floating-point input data and floating-point kernelcoefficients.

The multiplier multiplies 1302 the mantissas of the input data andkernel coefficients. The exponent adder adds 1304 the exponents togenerate an exponent value. The exponent adder modifies 1306 theexponent value using offset value from the shift register offset tomatch the multiplied value with a fixed-point precision of theaccumulator circuit.

The shift register generates 1308 the multiplied value by realigning themantissa value based on the exponent value. The adder adds 1310 themultiplied value and the accumulator value from the accumulator togenerate output data.

The output data is stored 1312 in the accumulator. The accumulatorprovides 1314 the output data to MAD as accumulated value for anotherprocessing cycle.

While particular embodiments and applications have been illustrated anddescribed, it is to be understood that the invention is not limited tothe precise construction and components disclosed herein and thatvarious modifications, changes and variations which will be apparent tothose skilled in the art may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope of the present disclosure.

What is claimed is:
 1. A neural processor circuit for acceleratingprocessing of neural networks, comprising: a neural engine circuit,including: a multiply-add (MAD) circuit comprising a shift register andan exponent adder that are used in a floating point mode, the MADcircuit configured to: receive fixed point input data and a fixed pointkernel coefficient in a fixed point mode; deactivate the shift registerand the exponent adder; and multiple the fixed point input data and thefixed point kernel coefficient to generate a multiplied value; and anaccumulator circuit coupled to the MAD circuit, the accumulator circuitconfigured to generate an output data by accumulating one or moremultiplied values generated by the MAD circuit.
 2. The neural processorcircuit of claim 1, wherein the neural engine circuit is configured toreceive a first input data that is represented as floating point data inthe floating point mode, and a second input data that is represented asfixed point data in a fixed point mode.
 3. The neural processor circuitof claim 1, wherein: the neural engine circuit includes a plurality ofMAD circuit coupled to the accumulator circuit; a first subset of theMAD circuits is configured to generate first output data of a firstchannel using an input data and a first kernel coefficient; and a secondsubset of the MAD circuits is configured to generate second output dataof a second channel using the input data and a second kernelcoefficient.
 4. The neural processor circuit of claim 3, wherein theneural engine circuit further comprises: a kernel extract circuitconfigured to: provide the first kernel coefficient to the first subsetof the MAD circuits; provide the second kernel coefficient to the secondsubset of the MAD circuits; and an input buffer circuit configured toprovide the input data to the first subset of the MAD circuits and thesecond subset of the MAD circuits.
 5. The neural processor circuit ofclaim 4, wherein the input buffer circuit comprises a shifter thatshifts read locations of the input buffer circuit to change portions ofthe input data selected to be send to the MAD circuits.
 6. The neuralprocessor circuit of claim 1, wherein the neural engine circuit isconfigurable to operate in a floating point mode, and wherein the MADcircuit, in the floating point mode is configured to modify an exponentvalue of a second multiplied value using an offset value to match themultiplied value to a fixed point precision.
 7. The neural processorcircuit of claim 6, wherein the MAD circuit includes a multiplier, inthe floating point mode, configured to multiply a mantissa of the inputdata and the mantissa of the kernel coefficient to generate a mantissavalue.
 8. The neural processor circuit of claim 6, wherein the shiftregister in the floating point mode is activated, the shift offsetregister configured to provide the offset value to the exponent adder,and the exponent adder configured to modify the exponent value using theoffset value to match the second multiplied value.
 9. The neuralprocessor circuit of claim 1, further comprising a post-processorcircuit, and wherein the accumulator selectively provides the outputdata to the MAD circuit or the post-processor circuit.
 10. The neuralprocessor circuit of claim 1, wherein the MAD circuit includes: amultiplier configured to multiply an input data and a kernel coefficientto generate the multiplied value; and an adder coupled to the multiplierand the accumulator circuit, the adder configured to add the multipliedvalue with an accumulated value.
 11. A method of operating a neuralprocessor circuit for accelerating processing of neural networks, themethod comprising: receiving fixed point input data and a fixed pointkernel coefficient in a fixed point mode of a multiply-add (MAD)circuit, the MAD circuit comprising a shift register and an exponentadder that are used in a floating point mode; deactivating the shiftregister and the exponent adder; multiplying the fixed point input dataand the fixed point kernel coefficient to generate a multiplied value;and generating, at an accumulator circuit coupled to the MAD circuit, anoutput data by accumulating one or more multiplied values generated bythe MAD circuit.
 12. The method of claim 11, further comprising:receiving a first input data that is represented as floating point datain the floating point mode, and a second input data that is representedas fixed point data in a fixed point mode.
 13. The method of claim 11,further comprising: generating, by a first subset of the MAD circuits,first output data of a first channel using an input data and a firstkernel coefficient; and generating, by a second subset of the MADcircuits, second output data of a second channel using the input dataand a second kernel coefficient.
 14. The method of claim 13, furthercomprising: storing the input data to the first subset of the MADcircuits and the second subset of the MAD circuits in an input buffer;providing the first kernel coefficient to the first subset of the MADcircuits; and providing the second kernel coefficient to the secondsubset of the MAD circuits.
 15. The method of claim 14, furthercomprising shifting read locations of the input buffer circuit to changeportions of the input data selected to be send to the MAD circuits. 16.The method of claim 11, wherein the neural processor circuit isconfigurable to operate in a floating point mode, and wherein the MADcircuit, in the floating point mode is configured to modify an exponentvalue of a second multiplied value using an offset value to match themultiplied value to a fixed point precision.
 17. The method of claim 16,further comprising, in the floating point mode, multiplying a mantissaof the input data and the mantissa of the kernel coefficient to generatea mantissa value.
 18. An electronic device, comprising: system memoryconfigured to store a neural network; and a neural processor circuit foraccelerating processing of the neural network, the neural processorcircuit comprising a multiply-add (MAD) circuit and an accumulator, theMAD circuit comprising a shift register and an exponent adder that areused in a floating point mode, the neural processor circuit configuredto: receive fixed point input data and a fixed point kernel coefficientin a fixed point mode; deactivate the shift register and the exponentadder; multiple the fixed point input data and the fixed point kernelcoefficient to generate a multiplied value; and generate an output databy accumulating one or more multiplied values generated by the MADcircuit.
 19. The electronic device of claim 18, the neural processorcircuit includes a plurality of MAD circuit coupled to the accumulatorcircuit; a first subset of the MAD circuits is configured to generatefirst output data of a first channel using an input data and a firstkernel coefficient; and a second subset of the MAD circuits isconfigured to generate second output data of a second channel using theinput data and a second kernel coefficient.
 20. The electronic device ofclaim 18, wherein the neural processor circuit is configurable tooperate in a floating point mode, and wherein the MAD circuit, in thefloating point mode is configured to modify an exponent value of asecond multiplied value using an offset value to match the multipliedvalue to a fixed point precision.